System and method for interactive haptic positioning of a medical device

ABSTRACT

A combination of a haptic device and a computer-assisted medical system is used for interactive haptic positioning of a medical device coupled to the haptic device. A reconfigurable haptic object facilitates the positioning of the medical device and/or the haptic device. The haptic object may be modified in response to application of a force against the haptic object by a user of the haptic device pushing the haptic device against the haptic object. Preferably, the haptic object moves in the direction of the force applied by the haptic device. The medical device may be guided to a desired pose relative to a target area from its current position. The user may approach the target area from its current position and still be provided with haptic cues to enable the user to guide the medical device to the target area.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application claims the benefit of Provisional PatentApplication Ser. No. 60/362,368, entitled “Haptic System for MedicalProcedures”, filed on Mar. 6, 2002, the disclosure of which isincorporated herein by reference. The following applications, filed on adate even herewith, are incorporated herein by reference: attorneydocket number 4204.17-1, entitled “System And Method For Intra-OperativeHaptic Planning Of A Medical Procedure”; attorney docket number 4204.18,entitled “System And Method For Using A Haptic Device As An InputDevice”; and attorney docket number 4204.19-1, entitled “System AndMethod For Haptic Sculpting Of Physical Objects”.

TECHNICAL FIELD OF THE INVENTION

The present invention relates generally to computer-assisted surgerysystems and surgical navigation systems, and more particularly to asystem and method for interactive haptic positioning of a medical deviceusing a reconfigurable haptic object.

BACKGROUND OF THE INVENTION

The functions of a computer-assisted surgery (CAS) system may includepre-operative planning of a procedure, presenting pre-operativediagnostic information and images in useful formats, presenting statusinformation about a procedure as it takes place, and enhancingperformance. The CAS system may be used for procedures in traditionaloperating rooms, interventional radiology suites, mobile operating roomsor outpatient clinics. Many approaches to CAS have been attemptedcommercially. The procedure may be any medical procedure, whethersurgical or non-surgical.

Navigation systems are used to display the positions of surgical toolswith respect to pre- or intraoperative image datasets. These imagesinclude intraoperative images, such as two-dimensional fluoroscopicimages, and preoperative three dimensional images generated using, forexample, magnetic resonance imaging (MRI), computer tomography (CT) andpositron emission tomography (PET). The most popular navigation systemsmake use of a tracking or localizing system. These systems locatemarkers attached or fixed to an object, such as an instrument or apatient, and track the position of markers. These tracking systems areoptical and magnetic, but also include acoustic systems. Optical systemshave a stationary stereo camera pair that observes passive reflectivemarkers or active infrared LEDs attached to the tracked tools. Magneticsystems have a stationary field generator that emits a magnetic fieldthat is sensed by small coils integrated into the tracked tools. Thesesystems are sensitive to nearby metal objects.

While navigation systems are relatively easy to integrate into theoperating room, a fundamental limitation is that they have restrictedmeans of communication with the surgeon. Most systems transmitinformation to the surgeon via a computer monitor. Conversely, thesurgeon transmits information to the system via a keyboard and mouse,touchscreen, voice commands, control pendant, or foot pedals, and alsoby moving the tracked tool. The visual displays of navigation systemsmay at best display multiple slices through three-dimensional diagnosticimage datasets, which are not easy to interpret for complex 3-Dgeometries. These displays also require the surgeon to focus his visualattention away from the surgical field.

When defining a plan using a tracked tool, it can be difficult tosimultaneously position the tool appropriately in multiple degrees offreedom (DOFs). Similarly, when aligning a tracked instrument with aplan, it is difficult to control the position of the tool in multiplesimultaneous DOFs, especially where high-accuracy is desirable. It isperhaps not a coincidence that navigation systems have had their largestacceptance in cranial neurosurgery, where most applications involvespecifying a trajectory to a feature of interest without hittingcritical features. Often, the tip of the tool is pressed against theanatomy and pivoted, effectively decoupling the position and orientationplanning of the trajectory.

Autonomous robots have been applied commercially to joint replacementprocedures. These systems make precise bone resections, improvingimplant fit and placement relative to techniques that rely on manualinstruments. Registration is performed by having the robot touchfiducial markers screwed into the bones or a series of points on thebone surfaces. Cutting is performed autonomously with a high-speed burr,although the surgeon can monitor progress and interrupt it if necessary.Bones must be clamped in place during registration and cutting, and aremonitored for motion, which then requires re-registration. Deficienciesreported by users of these systems include the large size of the robot,poor ergonomics, the need for rigidly clamping the bone for the 45-60minutes required for registration and cutting, and the need forincreasing the incision by 50-100 mm to provide adequate access for therobot. Furthermore, autonomous robots generally function best in highlystructured environments, as evidenced by the rigid clamping of the bonesof interest and making larger incisions to keep soft tissue away fromthe robot.

Except for specific steps of some surgical procedures, modern surgeriesdo not tend to provide well-structured environments for autonomousrobots. A robot is generally not able to keep track of the surgicalstaff and instrumentation required to support a procedure. Althoughstrict management of the operating environment might make this possible,the complexity of the human body will always provide a high degree ofunstructuredness.

Robotic technology can also be used to improve upon standard practicewithout requiring autonomous operation. Notable commercial systems ofthis type include teleoperated robotic systems for laproscopic surgeriesranging from gall-bladder removal to closed-chest beating heart coronarysurgery. These systems provide a console for the surgeon that includes ahigh-fidelity display and a master input device. The slave robot iscoupled to the master and physically interacts with the anatomy. Thebenefits of these systems are primarily in providing an ergonomicworking environment for the surgeon while improving dexterity throughmotion scaling and tremor reduction. Although the master console wouldnormally be in the same room as the patient, an interesting byproduct ofthese systems is that they enable telesurgery. However, the robots haveminimal autonomy in these systems, which is not surprising given thecomplexity involved in manipulating and altering soft tissue.

SUMMARY OF THE INVENTION

One aspect of the invention generally pertains to the combination of ahaptic device with a computer-assisted medical system. The combinationof a haptic device and a computer-assisted medical system may be usedfor interactive haptic positioning of the haptic device, optionallycoupling a medical device, at a desired position, for a medicalprocedure. A reconfigurable (or steerable) haptic object facilitatesinteractive haptic positioning of the medical device and/or the hapticdevice. The reconfigurability of the haptic object allows the hapticobject to be modified in response to application of a force against thehaptic object by a user of the haptic device pushing the haptic deviceagainst the haptic object. Preferably, the haptic object moves in thedirection of the force applied by the haptic device. Because the hapticobject is reconfigurable, the haptic device and/or the medical devicemay be guided to a desired pose relative to a target area from itscurrent position or pose. The user may approach the target area from itscurrent position without colliding with other medical tools andequipment, the patient or operating room staff, and still be providedwith haptic cues to enable the user to guide the medical device to thetarget area.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, the objectsand advantages thereof, reference is now made to the followingdescriptions taken in connection with the accompanying drawings inwhich:

FIG. 1 is a diagrammatic illustration of an exemplary operating room inwhich a haptic device is used with a computer-assisted surgery system;

FIG. 2 illustrates an exemplary haptic device being used in conjunctionwith a computer-assisted surgery system;

FIGS. 3A and 3B illustrate different types of haptic objects;

FIG. 3C is a flowchart of an exemplary method for intra-operative hapticplanning of a surgical procedure;

FIG. 4A illustrates the use of a dynamic haptic object for placement ofa haptic device;

FIG. 4B is a flowchart of a method for interactive haptic positioning ofa medical device coupled to a haptic device;

FIG. 5 illustrates the use of an exemplary haptic device in conjunctionwith a computer-assisted surgery system;

FIG. 6A illustrates an exemplary haptic device being used for hapticsculpting of physical objects;

FIG. 6B illustrates an exemplary haptic object for haptic sculpting ofphysical objects;

FIG. 6C is a flowchart of a method for dynamically modifying a hapticobject;

FIGS. 7A and 7B illustrate the use of an exemplary haptic device and asurgical tool to define a haptic object;

FIG. 8 illustrates the use of an exemplary haptic device as an inputdevice; and

FIG. 9 is a flowchart of a representative method for using a hapticdevice as an input device.

DETAILED DESCRIPTION OF THE DRAWINGS

In the following description, like numerals refer to like elements.References to “surgeon” include any user of a computer-assisted surgicalsystem, a surgeon being typically a primary user. References to“surgical procedure” include any medical procedure, whetherinterventional or non-interventional, an interventional procedure beingtypically the primary procedure.

A haptic device is a mechanical or electro-mechanical device thatinteracts and communicates with a user, such as a surgeon, using sensoryinformation such as touch, force, velocity, position, and/or torque.Some robots may be used as haptic devices, though haptic devices mayinclude devices that are not necessarily considered to be robots in aconventional sense. Haptic devices typically have little autonomy.

In general, a component of interest may be optionally coupled to thehaptic devices. A component of interest may comprise a medical device,for example a surgical tool, a microscope, a laser range finder, acamera, a surgical light, an endoscope, an ultrasound probe, aradiotherapy device, interventional medical tools, rehabilitativesystems for physical therapy, and/or the like. The terms “medicaldevice”, “surgical device” and “surgical tool” are used interchangeablyherein.

For example, when used during surgery, such devices cooperatively hold asurgical instrument in conjunction with the surgeon. The surgeon movesthe surgical instrument with the assistance of, or input from, thehaptic device. Alternatively, in a teleoperation system, the hapticdevice may exclusively hold the surgical instrument. In such animplementation, the surgeon moves a “master” haptic device that iscoupled to a “slave” device in order to interactively manipulate thesurgical tool. In a teleoperation system, the master haptic device maybe physically separated from the surgical site to provide a moreergonomic or immersive working position for the surgeon and/or allow thesurgeon to perform the surgery remotely. In an impedance mode, a hapticdevice measures or senses the pose (position, orientation, velocity,and/or acceleration) of the surgical instrument and applies forcesand/or torques (“wrench”) to the instrument. In an “admittance” mode, ahaptic device measures the wrench at some location on the device (orsurgical instrument) and acts to modify the position of the instrument.There may be a static, quasi-static, or dynamic mapping between thesensed pose and output wrench. Common mappings may include wrenches thatresult from the tool interacting with “virtual” objects defined by orwith input from a user, which may include mathematical or simulatedmechanical constraints.

A “haptic object” is used herein to describe such a mapping. In somecases, a haptic object may only produce non-zero outputs for certainjoint angles of the haptic device, or only for certain endpointpositions and/or orientations of the haptic device. A haptic object maybe a smoothly time varying mapping and/or may only exist for certaintimes. A haptic object may have an associated spatial or geometricrepresentation that corresponds to locations where the mapping isdiscontinuous or has other properties that can be felt by the user wheninteracting with the haptic object. For example, if a haptic object onlyproduces non-zero outputs when the endpoint of the haptic device lieswithin a spherical region in space, then it may be useful to present acorresponding spherical representation to the user. However, a hapticobject may not necessarily have such a clearly defined boundary orsimilar internal structures. A haptic object may be active over theentire range of endpoint positions, endpoint orientations, and/or jointpositions of the haptic device or only a portion of these ranges. Theremay be multiple haptic objects active at any given time, possibly inoverlapping portions of space.

A “haptic cue” is used to describe an aspect of the mapping of a hapticobject. Having a cue may convey information or produce a desired effectwhen the user interacts with the haptic object. Haptic cues and hapticobjects do not necessarily correspond to user interface or softwareprogramming components in a particular embodiment and may be simply oneof many ways to design, implement, present to the user the mappingsbetween the inputs and outputs of the haptic device.

The reduction or elimination of autonomy increases the comfort level ofusers, such as surgeons. Any time a robot moves autonomously, thesurgeon is no longer in control and must simply observe the robot'sprogress. Robot motions have to be slow to provide adequate time for thesurgeon to respond should something unusual happen. If, however, a robotacts, at least mostly, in a passive manner, even if capable of activemotions, then the surgeon does not cede control to the robot.

Using a device capable of active motions in such a way that it only actslike a passive device from the user's perspective has advantages. Activeactuators can be used to counteract the effect of gravity, allowing agreater variety of mechanism designs. The device can be used in anautonomous mode for performing automated testing and service procedures.

FIG. 1 is a diagrammatic illustration of an exemplary operating room inwhich a haptic device 113 is used with a computer-assisted surgerysystem 11. Computer-assisted surgery system 11 comprises a displaydevice 30, an input device 34, and a processor based system 36, forexample a computer. Input device 34 may be any input device now known orlater developed, for example, a keyboard, a mouse, a trackball, and/orthe like. Display device 30 may be any display device now known or laterdeveloped for displaying two-dimensional and/or three-dimensionalimages, for example a monitor, a wearable display, a projection display,a head-mounted display, stereoscopic views, a display device capable ofdisplaying image(s) projected from an image projecting device, forexample a projector, and/or the like. If desired, display device 30 maybe a display device capable of displaying a holographic image. Ifdesired, display device 30 may be a touch screen and be used as an inputdevice.

Haptic device 113 is, in the illustrated example, a robotic device.Haptic device 113 may be controlled by a processor based system, forexample a computer 10. Computer 20 may also include power amplificationand input/output hardware. Haptic device 113 may communicate withcomputer-assisted surgery system 11 by any communication mechanism nowknown or later developed, whether wired or wireless.

Also shown in FIG. 1 is a storage medium 12 coupled to processor basedsystem 36. Storage medium 12 may accept a digital medium which storessoftware and/or other data. A surgical tool or instrument 112 is showncoupled to haptic device 113. Surgical tool 112 is preferablymechanically coupled to haptic device 113, such as by attaching orfastening it. However, if desired, surgical tool 112 may be coupled,either directly or indirectly, to haptic device 113 by any other method,for example magnetically. If desired, vacuum may be used to couplesurgical tool 112 to haptic device 113. Surgical tool 112 may behaptically controlled by a surgeon remotely or haptically controlled bya surgeon 116 present in proximity to surgical tool 112.

Haptic object 110 is a virtual object used to guide and/or constrain themovement and operations of surgical tool 112 to a target area inside apatient's anatomy 114, for example the patient's leg. In this example,haptic object 110 is used to aid the surgeon to target and approach theintended anatomical site of the patient. Haptic feedback forces are usedto slow and/or stop the surgical tool's movement if it is detected thata portion of surgical tool 112 will intrude or cross over predefinedboundaries of the haptic object. Furthermore, haptic feedback forces canalso be used to attract (or repulse) surgical tool 112 toward (or awayfrom) haptic object 110 and to (or away from) the target. If desired,surgeon 116 may be presented with a representation of the anatomy beingoperated on and/or a virtual representation of surgical tool 112 and/orhaptic object 110 on display 30.

When surgical tool 112 is haptically controlled by a surgeon remotely,for example when conducting a teleoperation, the surgeon controls themovement of the surgical tool using the master haptic device and/or areal or simulated display of the surgical tool, patient anatomy, and/oradditional haptic or visual objects designed to aid the surgicalprocedure. Haptic feedback forces may be transmitted by slave hapticdevice 113 to the surgeon at the remote location via the master hapticdevice to guide the surgeon. Alternatively, the haptic feedback forcesmay be generated at the master device and transmitted to the surgeondirectly. In some cases either the slave or master device may be apositioning device with little or no haptic capabilities.

The CAS system preferably includes a localization or tracking systemthat determines or tracks the position and/or orientation of varioustrackable objects, such as surgical instruments, tools, haptic devices,patients, and/or the like. The tracking system continuously determines,or tracks, the position of one or more trackable markers disposed on,incorporated into, or inherently a part of the trackable objects, withrespect to a three-dimensional coordinate frame of reference. Markerscan take several forms, including those that can be located usingoptical (or visual), magnetic or acoustical methods. Furthermore, atleast in the case of optical or visual systems, location of an object'sposition may be based on intrinsic features, landmarks, shape, color, orother visual appearances, that, in effect, function as recognizablemarkers.

Any type of tracking system may be used, including optical, magnetic,and/or acoustic systems, that may or may not rely on markers. Presentday tracking systems are typically optical, functioning primarily in theinfrared range. They usually include a stationary stereo camera pairthat is focused around the area of interest and sensitive to infraredradiation. Markers emit infrared radiation, either actively orpassively. An example of an active marker is a light emitting diodes(LEDs). An example of a passive marker is a reflective marker, such asball-shaped marker with a surface that reflects incident infraredradiation. Passive systems require a an infrared radiation source toilluminate the area of focus. A magnetic system may have a stationaryfield generator that emits a magnetic field that is sensed by smallcoils integrated into the tracked tools.

With information from the tracking system on the location of thetrackable markers, CAS system 11 is programmed to be able to determinethe three-dimensional coordinates of an end point or tip of a tool and,optionally, its primary axis using predefined or known (e.g. fromcalibration) geometrical relationships between trackable markers on thetool and the end point and/or axis of the tool. A patient, or portionsof the patient's anatomy, can also be tracked by attachment of arrays oftrackable markers. In the illustrated example, the localizer is anoptical tracking system that comprises one or more cameras 14 thatpreferably track a probe 16. As shown in FIG. 1, cameras 14 may becoupled to processor based system 36. If desired, cameras 14 may becoupled to computer 10. Probe 16 may be a conventional probe now knownor later developed. If desired, the probe may be rigidly attached tohaptic device 113 or integrated into the design of haptic device 113.

If desired, in an implementation, processor based system 36 may comprisea portion of image guided surgery software to provide minimal userfunctionality e.g., retrieval of previously saved surgical information,preoperative surgical planning, determining the position of the tip andaxis of instruments, registering a patient and preoperative and/orintraoperative diagnostic image datasets to the coordinate system of thetracking system, etc. Image guided surgery using this method may not bepossible with the computer alone. As such, full user functionality maybe enabled by providing the proper digital medium to storage medium 12coupled to computer 36. The digital medium may comprise an applicationspecific software module. The digital medium may also comprisedescriptive information concerning the surgical tools and otheraccessories. The application specific software module may be used toassist a surgeon with planning and/or navigation during specific typesof procedures. For example, the software module may display predefinedpages or images corresponding to specific steps or stages of a surgicalprocedure. At a particular stage or part of a module, a surgeon may beautomatically prompted to perform certain tasks or to define or enterspecific data that will permit, for example, the module to determine anddisplay appropriate placement and alignment of instrumentation orimplants or provide feedback to the surgeon. Other pages may be set upto display diagnostic images for navigation and to provide certain datathat is calculated by the system for feedback to the surgeon. Instead ofor in addition to using visual means, the CAS system could alsocommunicate information in ways, including using audibly (e.g. usingvoice synthesis) and tactilely, such as by using a haptic interface ofdevice. For example, in addition to indicating visually a trajectory fora drill or saw on the screen, a CAS system may feedback to a surgeoninformation whether he is nearing some object or is on course with anaudible sound. To further reduce the burden on the surgeon, the modulemay automatically detect the stage of the procedure by recognizing theinstrument picked up by a surgeon and move immediately to the part ofthe program in which that tool is used.

The software module may be such that it can only be used a predefinednumber of times. If desired, the software module functions only whenused in conjunction with the portion of the image guided surgerysoftware that resides on computer 36. The software which resides oncomputer 36 in conjunction with the software on the digital mediumprocesses electronic medical diagnostic images, registers the acquiredimages to the patient's anatomy, and/or registers the acquired images toany other acquired imaging modalities, e.g., fluoroscopy to CT, MRI,etc. if desired, the image datasets may be time variant, i.e. imagedatasets taken at different times may be used. Media storing thesoftware module can be sold bundled with disposable instrumentsspecifically intended for the procedure. Thus, the software module neednot be distributed with the CAS system. Furthermore, the software modulecan be designed to work with specific tools and implants and distributedwith those tools and implants. Moreover, CAS system can be used in someprocedures without the diagnostic image datasets, with only the patientbeing registered. Thus, the CAS system need not support the use ofdiagnostic images in some applications i.e. an imageless application.

An example of the illustrated robotic arm is a robotic arm manufacturedby Barrett Technology, and referred to as the “Whole-Arm Manipulator” or“WAM”. This robotic arm has a cable transmission, which provides highbandwidth, backdrivability, and force fidelity. However, other roboticdevices capable of impedance or admittance modes of haptic interactioncould be used. For example, direct-drive systems or systems with othertypes of low-friction transmissions or systems with a combination oftransmission types may also be well-suited to serve as a haptic devicefor surgical applications. Furthermore, the haptic device need notnecessarily take the form of a robotic arm. The WAM robotic arm has afour degrees of freedom of movement. However, it is augmented by a 1-DOFdirect-drive wrist for trajectory-based medical applications. Ifdesired, degrees of freedom may be added or removed without affectingthe scope of the illustrated invention.

Though it has some advantages, a cable transmission has somedisadvantages. It requires careful installation and maintenance toprevent the possibility of failure during a procedure. Furthermore, acable transmission is not as stiff as geared transmissions. Similardeficiencies may also be found in haptic devices using other types oftransmissions.

These deficiencies may be addressed by augmenting existing positionsensors that are mounted on drive motors with additional redundantsensors. These sensors may be of various types, including withoutlimitation rotary encoders or resolvers, tilt sensors, heading (compass)sensors, sensors that detect the direction of gravity, an optical,magnetic or acoustical tracking system (such as optical camera systemsof the type commonly used to track surgical instruments), or laser-basedposition sensing. The output of these sensors can be compared with theoriginal sensors to detect discrepancies that may indicate problems inthe transmissions or sensors. In addition, the added sensors can be usedto detect both low bandwidth deflections in the cable transmissions,which the system can then easily compensate for using well-known controltechniques. The sensor may also detect the high bandwidth deflections inthe cable transmissions, which can provide an additional input to theservo loop and permit improved stability of the servo system, usingwell-known control techniques for systems that include sensors on boththe drive and load sides of a transmission. The sensor can also improvethe accuracy of the determination of the pose of the arm by reducing oreliminating the effect of deflections of the arm links and/ortransmission. Such sensors could also be used to overcome similardeficiencies in robotic devices using other types of transmissionsystems.

When performing surgery, a haptic device capable of holding a tool, e.g.a drill guide or other similar constraint or attachment mechanism forsurgical tools is positioned relative to the patient such that it canattain the poses appropriate for a variety of approaches for aparticular procedure. It is also registered to the physical anatomy suchthat it can correlate information in diagnostic or planning imagedatasets, which can be two or three dimensional, to locations inphysical space using well-known registration techniques. The imagedatasets may be one or more images generated using for example, magneticresonance imaging (MRI), computer tomography (CT), positron emissiontomography (PET), magnetic resonance angiography (MRA), single photonemission computed tomography (SPECT), magnetic resonance venography(MRV), contrast enhanced MR venography (CEMRV), CT angiography, CTmyelography, MR angiography, MR myelography, fluoroscopy, opticalimaging, isotope imaging, ultrasound microscopy, laproscopic ultrasound,and MR spectrometry. Such images may include, for example, x-ray images,digital x-ray images, computer tomography images, MRI images, MRAimages, MR spectrometric images, PET images, MRV images, SPECT images,CEMRV images, CT angiographic images, CT myelographic images, MRmyelographic images, flair images, two-dimensional fluoroscopic images,three-dimensional fluoroscopic images, two-dimensional ultrasonicimages, three-dimensional ultrasonic images, ultrasound microscopyimages, laproscopic ultrasound images, optical images, isotopic images,laser depth maps, line arts, sketches, “cartoon” representations,holographic images, and/or the like.

Features to be avoided, such as blood vessels, tendons, nerves, andcritical areas of the brain can be automatically, semi-automatically, ormanually defined on the image datasets. Features targeted by theprocedure, such as tumors, osteophytes, anatomical targets fordeep-brain stimulation, biopsy sites, anatomical sites for implantplacement, or other regions of the anatomy can also be automatically,semi-automatically, or manually defined on the image datasets.

The image dataset(s), coupled with definitions of features to beavoided, can be used to create haptic “cues” that indicate to thesurgeon that a violation of sensitive anatomy is taking place. A generalfunction of these types of cues is to apply forces and/or torques thattend to repulse the haptic device from poses where an instrumentattached to the device would, for example, impact the defined criticalfeatures. Similarly, the image dataset(s), coupled with the definitionsof features to be targeted can also used to create haptic cues thatindicate to the surgeon that the desired target region would be reachedby the surgical instrument appropriately attached to the haptic arm. Ageneral function of these types of cues is to attract the haptic deviceto such poses or lock the haptic device into these poses once they areattained.

While the haptic device can be deployed as a fully integrated componentof a computer-aided surgery system, there are advantages to having thehaptic device act as an optional peripheral to such a system. The systemis then convenient to use for procedures that do not require the use ofthe haptic device. There arc also development and architecturaladvantages to this approach. The haptic device will likely require areal-time operating system or special motion control hardware togenerate high-frequency updates for the haptic control system. Thecomputer-aided surgery system will have different requirements, such asfast graphics processing hardware and compatibility requirements with arange of user input and output devices, so that there are advantages ofhaving two computer systems to meet the differing uses. Separating thecomputer surgery and haptic arm components also has safety advantages.The haptic device therefore preferably contains only computing softwareand hardware that is necessary for ensuring high-performance, stable,and safe operation. The computer aided surgery system can containsoftware and hardware for connecting to a hospital network, displayingvarious graphical views, supporting various user input/output devices,managing libraries of implant and instrument databases, and/or any otherfunctionality useful in such a system. This architecture also allowsdevelopers with minimal knowledge of haptic systems to buildapplications that use the haptic device. The physical interface betweenthese two systems can be wired or wireless, such as a serial, USB, orother cable communications interface, or wireless ethernet, wirelessserial, infra-red or other wireless communications system. The softwareinterface between these systems would include a set of commands thatallows the computer aided surgery system to control operation of thehaptic device. For example, the computer-aided surgery system may send acommand to the haptic device requesting it to enter into a joystick-likeinput mode with certain stiffness parameters. The haptic arm systemchecks if the parameters are safe and otherwise acceptable, and thenenters into such a mode or responds with an appropriate error message.The computer-aided surgery system and haptic device may also beintegrated into a single system unit, or may be implemented using asingle or a multi-processor computing device. The CAS system, the hapticdevice and/or computer 10 may also be integrated into another piece ofequipment, such as an imaging equipment (e.g., fluoroscopy, CT, MR,ultrasound, and/or the like), an equipment cart in the room where themedical procedure is performed, and/or the like.

Referring to FIG. 2, representative “haptic object” 20 is atwo-dimensional virtual plane. However, it is only an example of hapticobjects generally, which may be zero (e.g. a point), one (e.g. a virtualline or path), two (e.g. a virtual plane or flat surface), or threedimensional (e.g. a virtual curved surface, a cube or other solidobject), and may have simple or complex geometric shapes. Haptic object20 is preferably defined with respect to the space of a physical object,such as patient anatomy 114. Haptic object 20 is defined to guide and/orconstrain the movement of haptic device 113. The distance between hapticdevice 113 and haptic object 20 is shown in FIG. 2 by X and the distancebetween patient's anatomy 114 and haptic object 20 is shown by X₁.Haptic object 20 may be used in connection with generating forcefeedback on haptic device 113. The generation of force feedback may alsodepend on various factors, for example, the velocity at which hapticdevice 113 is approaching patient's anatomy 114, the position of hapticdevice 113, haptic object 20, and/or the like. An algorithm whichcomputes the current position of haptic device 113 relative to hapticobject 20 may be used to provide information to the surgeon about thelocation of haptic device 113 relative to haptic object 20. When hapticdevice 113 comes within a predefined distance of haptic object 20, astiffness parameter may be changed to make it more difficult to movehaptic device 113. If desired, force may be applied in a direction awayfrom anatomy 114 to resist the movement of haptic device 113 towardanatomy 114 or to move haptic device 113 away from anatomy 114.

It may not be appropriate to implement rigid haptic objects, such asvirtual surfaces and walls, in certain cases. A surgeon will lose theability to feel the anatomy in any direction that is rigidly constrainedby the haptic device. In many applications, precise localization ofanatomical features cannot be achieved by simply combining diagnosticdatasets with a tool tracking system or precision robotic devices.Changes in the anatomy after the diagnostic datasets are taken, unsensedmotion in the kinematic chain connecting the anatomical features ofinterest and the tracking system's camera or haptic device, registrationerrors, and inaccuracies in the localization devices will contribute topositioning errors. Although CAS systems may be used to position thesurgical tool very close to the target region, more accurate positioningis often difficult or prohibitively costly. In some medical procedures,such as pedicle screw placement in the upper thoracic and cervicalportions of the spine, deep brain neurosurgical procedures, etc., aslight inaccuracy may adversely affect the medical procedure beingperformed. Therefore, it is desirable in these types of procedures thata surgeon retain an ability to feel the anatomy.

Haptic devices can be used for registering patients to CAS systems anddiagnostic data sets of the patient's anatomy, for example, by attachinga probe and touching it to a few selected anatomical landmarks,implanted fiducials, or multiple points on a surface of interest. Theycan be used for haptic exploration of diagnostic datasets to augment thevisual display of this information. This exploration may occurintra-operatively while registered to the actual patient anatomy orpre-operatively in a purely virtual way. This haptic exploration isespecially useful for exploring complex three-dimensional structures,where the surgeon's highly developed sense of touch can be used toexplore complexities or subtleties of the dataset that may be difficultor impossible to display adequately on a two-dimensional or eventhree-dimensional visual display.

While performing traditional freehand surgery, surgeons rely on localanatomical features to ensure proper positioning of the surgical tool.If the ability of the surgeon to feel the patient anatomy is preserved,the surgeon can explore the local anatomy and correct these localizationerrors based on his expert knowledge of structures of interest. In thisway, the final positioning is determined by nearby anatomical featuresrather than a tracking system sitting across the operating room or arobot whose base may not be rigidly connected to the patient.

A portion of surgical tool 112 coupled with a haptic device, for examplethe tip of surgical tool 112, may be used to sense properties of thelocal anatomy. The properties of the local anatomy may be used toposition surgical tool 112 or to verify the proper positioning ofsurgical tool 112. The properties that may be sensed or monitored by thetool include electrical properties of the anatomy, force, pressure,stiffness, conductivity, etc. The information from the tip may beprovided back to CAS system 11. The information may then, if desired, becorrelated with information from diagnostic image datasets of thepatient. If desired, information from the tool may be used to augment orreplace the information from the image datasets. In either case theinformation may be used for better placement of surgical tool 112.

Location or position information of the tool may be sensed and providedback to CAS system 11 without the use of a separate sensor. The surgeonmay manually move surgical tool 112 to the desired position. Positioninformation of the tip of surgical tool 112 in the desired position maybe determined directly by CAS system 11 and/or computer 10 without theuse of a separate sensor. Other properties of the anatomy may be sensedby placing sensors at the tip of surgical tool 112. The output from thesensors may be provided back to CAS system 11 for processing.

The collected information may be used for a variety of purposes, such asalerting the user to registration errors, fully or partially correctingregistration errors, displaying graphical representations of theinformation on display device 30, defining haptic objects to assist theuser, displaying graphical representations of the information on displaydevice 30 superimposed over one or more images of the anatomy, and/orthe like. If desired, the collected information may be logged for use inmachine learning techniques.

The combination of a haptic device and a CAS system is also useful forcombining haptic exploration of diagnostic datasets and use of thehaptic device as a primary input device for planning. In this way,haptic exploration naturally leads the user to a suitable plan forperforming a procedure. Additionally, in some circumstances it ispossible to have the haptic device and the tool coupled with it in thecorrect position for performing a procedure as a result of thisexploration/planning process, eliminating the need to move the hapticdevice into position as a separate step.

Referring to FIG. 3A, it may be desirable in certain procedures toconfine the surgical instrument to a small working volume, in which caseit may stay within a working area inside a haptic object during theentire procedure. It may be necessary in certain cases to segment ordefine manually certain important features, but for most applicationsautomated segmentation of the diagnostic datasets will be sufficient forproviding appropriate haptic feedback.

In the illustrated embodiment, one or more attractive haptic objects areassociated with a target region for performing the surgical procedureand one or more repulsive haptic objects are associated with anatomicalfeatures to be avoided during the surgical procedure. For example, asshown in FIG. 3A, haptic object 22 defines a working area or volume forconstraining movement of surgical tool 112. On the other hand, as shownin FIG. 3B, haptic object 24 defines a working area or volume forconstraining movement of surgical tool 112 so that it is prevented fromcoming close to critical regions, such as nerves 25, organs 27, etc. Forexample, once the haptic objects are defined, the user performs surgicalplanning by pushing haptic device 113 around until a pose is found wherethe cues from the attractive haptic objects are active indicating thatsurgical tool 112, when attached to haptic device 113, would reach thetarget region, and where the cues from the repulsive haptic objects areinactive, indicating that surgical tool 112 would not penetrate any ofthe defined sensitive anatomical regions. In most cases, theserequirements will not fully constrain the pose of the arm and the usercan move the arm within this range of acceptable approaches based on anysecondary criteria a user finds appropriate. In some cases, the arm mayachieve an equilibrium state where multiple attractive or repulsivehaptic cues act in opposite directions. The user might mistake thisconfiguration to be an acceptable pose, even though the target regionmight not be reached or the critical anatomy regions might be violated.The user may be alerted to this situation in a number of ways, includingaudible or visual indicators, or by a haptic cue such as a vibration ofhaptic device 113. The user could then correct this situation by pushingthe haptic device away from this pose. Once in a pose satisfactory tothe user, haptic device 113 can be locked into position, using hardwarebrakes, control servoing techniques, or any other appropriate method toprovide a stable physical reference for the surgical procedure.

If fine adjustments are desired, the haptic device can be operated usinga mode where motion scaling, constraints, or other methods are used tomake such corrections that might otherwise be beyond the dexterity ofthe surgeon. For example, a control servo can be enabled to lock thedevice to a certain finite stiffness at the approximate desired pose.The surgeon can then make fine adjustments to this pose using a varietyof methods. For example, the surgeon may use a touch screen, a keyboard,a mouse, a trackball or voice inputs. If desired, the surgeon may pushthe end of the haptic device in the desired direction. In response tothese inputs, the system would adjust the desired pose appropriately,possibly in small increments that would be difficult to achieve bydirect positioning of the haptic device. It may be desirable to lockonly a portion of the pose so that the surgeon can focus on a morelimited number of adjustments at one time. This fine adjustment mayoccur after the coarse haptic positioning is complete, simultaneous withthe coarse haptic positioning, or interleaved with the coarse hapticpositioning.

For example, selecting a trajectory for a cranial neurosurgicalprocedure such as a biopsy, tumor resection, or deep-brain stimulationis a complicated 3-D planning problem. The surgeon must find a path to atarget area while avoiding blood vessels and sensitive areas of thebrain. If these regions can be turned into repulsive haptic objects,planning such a procedure may be as simple as applying a hapticconstraint that keeps the trajectory of a tool guide passing through thetarget of interest, and allowing the user to pivot the device about thispoint until it settles into a suitable pose where none of the repulsivehaptic objects are violated.

FIG. 3C is a flowchart of an exemplary method 140 for intra-operativehaptic planning of a surgical procedure. Haptic device 113 is placed inthe operating room such that surgical tool 112 may be positioned over alarge portion of a clinically reasonable range of surgical approachesfor a given surgical procedure. Surgical planning using method 140 isperformed in the presence of the patient and preferably without surgicaltool 112 being coupled to haptic device 113. Surgical tool 112 may be anon-contact medical device, such as a diagnostic or therapeuticradiation source. If desired, surgical planning using method 140 may beperformed with surgical tool 112 coupled to haptic device 113 but beingin a retracted state. When surgical tool 112 comprises a non-contactmedical device, it is preferably in a disabled state. A representationof the anatomy of the patient to be operated on may be displayed ondisplay device 30 along with a “virtual tool”. The virtual tool may be ahigh-fidelity representation or a schematic representation of surgicaltool 112, such as an axis, a point, or other feature of surgical tool112. The virtual tool indicates relative to the anatomy of the patient,the position and/or angle of surgical tool 112 or some portion thereofif the surgical tool had been coupled to haptic device 113 in its normalor enabled state.

In step 142, haptic device 113 is registered to the anatomy of thepatient. If desired, the representation of the anatomy of the patientdisplayed on display device 30 may also be registered with the anatomyof the patient so that information in diagnostic or planning datasetsmay be correlated to locations in the physical space. Any method forregistering, now known or later developed, may be used. In step 144, thetarget region is defined. The target region may be, for example, atumor, an osteophyte, an anatomical target for deep-brain stimulation, abone channel, and/or the like. The target region may be defined in anymanner now known or later developed. For example, the user, such as thesurgeon, may manually identify the target region on display device 30.If desired, the surgeon may define the target region by touching one ormore points on the target region or circling the target region ondisplay device 30 with a tool. Alternatively, the surgeon may define thetarget region by pointing a tool mounting axis of haptic device 113 tothe target region or by using haptic device 113 as an input device.Preferably, the identified target region is automatically highlighted ondisplay device 30. The tool mounting axis of haptic device 113 may be ofany shape, for example curved, straight, and/or the like. Regardless ofthe manner in which the target region is defined, it is desirable thatonce defined, the target region be clearly displayed on display device30 for confirmation. One or more attractive haptic objects, such ashaptic object 22 of FIG. 3A, may be associated with the target region.

In step 146, anatomical obstacles to be avoided are defined. Theanatomical obstacles comprise features to be avoided during surgery,such as major blood vessels, tendons, nerves, critical areas of thebrain, organs, healthy bones or other tissues, and/or the like. Theanatomical obstacles may be defined in any manner now known or laterdeveloped. For example, the surgeon may manually identify the anatomicalobstacles on display device 30. If desired, the surgeon may define theanatomical obstacles by touching one or more points on the anatomicalobstacles or circling the anatomical obstacles on display device 30 witha tool. Alternatively, the surgeon may define the anatomical obstaclesby pointing the tool mounting axis of haptic device 113 to theanatomical obstacles or by using haptic device 113 as an input device.Preferably, the identified anatomical obstacles are highlighted ondisplay device 30. Regardless of the manner in which the anatomicalobstacles are defined, it is desirable that, once defined, theanatomical obstacles are clearly displayed on display device 30 forconfirmation. One or more repulsive haptic objects, such as hapticobject 24 of FIG. 3B, may be associated with the defined anatomicalobstacles. Preferably, each anatomical obstacle has one repulsive hapticobject associated with it, although if desired more than one repulsivehaptic object may be associated with an anatomical obstacle.

In step 148, haptic device 113 is positioned, preferably by the surgeon,such that if surgical tool 112 were coupled to haptic device 113 or ifsurgical tool 112 were in an operating state, then the appropriateportion of the surgical tool would have the desired relationship withthe target region. For example, when coupled to haptic device 113,surgical tool 112 would penetrate the target region. Surgical tool 112is in its operating state when it is coupled to haptic device 113 and isnot retracted and/or is not disabled. Step 148 is preferably performedwithout regard to whether or not the tool may intersect the anatomicalobstacles in this position. A virtual tool displayed on display device30 is such that it's position and orientation corresponds to theposition and orientation of surgical tool 112 if surgical tool 112 hadbeen mounted on haptic device 113 or if surgical tool 112 were in itsnormal operating state. Thus, the surgeon may position haptic device 113in the desired pose while viewing the display on device 30, such thatthe virtual tool has the appropriate relation with the target region.

In step 152, a determination is made as to whether the virtual tool isintersecting any anatomical obstacles. If the virtual tool is notintersecting any anatomical obstacles, then the process starting at step162 is executed. Otherwise, the process starting at step 154 isexecuted. In step 154, haptic cues are provided by haptic device 113 tothe user. The haptic cues may be provided to the user based on one ormore haptic objects, for example the attractive haptic object(s)associated with the target region and/or the repulsive haptic object(s)associated with the anatomical obstacles. The repulsive haptic object(s)generate forces and/or torques that guide haptic device 113 away fromposes where the virtual tool would intersect the anatomical obstacles.Preferably, the repulsive haptic cues are active when the virtual toolpenetrates the repulsive haptic objects or is in proximity to therepulsive haptic objects. The attractive haptic object(s) cause thehaptic device to generate forces and/or torques that guide haptic device113 toward poses where the virtual tool has the desired relationshipwith the target region.

It is possible that the position of haptic device 113 may be such thatcues from multiple haptic objects cancel each other out even though thevirtual tool may be violating the anatomical obstacles. As such, in step156, a determination is made as to whether haptic cues from multipleobstacles are canceling each other out. If haptic cues from multipleobstacles are not canceling each other out, then the process starting atstep 158 may be executed. If haptic cues from multiple obstacles arecanceling each other out, then in step 160, a special haptic cue, forexample a vibration, may be provided to alert the user of this situationand the process starting at step 158 may be executed.

In step 158, haptic device 113 is moved, preferably by the surgeon.Haptic device 113 is preferably moved based at least in part on thehaptic cues provided by haptic device 113 to the surgeon. The positionof surgical tool 112 had it been coupled to haptic device 113 is trackedby the virtual tool and displayed on display device 30. Preferably, theuser moves haptic device 113 until an equilibrium pose is found. In theequilibrium position, the cues created by the attractive haptic objectsare active and those created by the repulsive haptic objects areinactive. The process starting at step 152 may then be executed todetermine whether the virtual tool is intersecting any anatomicalobstacles.

In step 162, a determination is made as to whether the user is satisfiedwith the trajectory to the target region. The user may make thisdetermination by viewing the virtual tool relative to the target regionas illustrated on display device 30. If the user is not satisfied withthe position and/or the orientation of the virtual tool, then theprocess starting at step 158 may be executed. If the user is satisfiedwith the position and the orientation of the virtual tool relative tothe target region and the obstacles, then the process starting at step164 may be executed. The user may indicate its satisfaction in one ormore of a number of ways. For example, the user may issue a voicecommand to indicate that it is satisfied with the position andorientation of the virtual tool. If desired, the user may activate afoot pedal or a button associated with the computer-assisted surgerysystem or haptic device 113 to indicate its satisfaction. If desired,the user may indicate its satisfaction via a touch screen, a keyboard, amouse, and/or the like, associated with the computer-assisted surgerysystem or haptic device 113. In step 164, haptic device 113 may belocked in the current pose.

Once the pose of haptic device 113 is locked, the surgical procedure maybe performed, for example by coupling surgical tool 112 to haptic device113 or by placing surgical tool 112 in its fully functional oroperational configuration. Because the pose of surgical tool 112relative to the anatomy has already been determined with the aid of thevirtual tool, surgical tool 112 will achieve the desired position whenit is coupled to haptic device 113 or when it is configured for use.

The illustrated method for intra-operative haptic planning of a surgicalprocedure may be implemented in software, hardware, or a combination ofboth software and hardware. The steps discussed herein need not beperformed in the stated order. Several of the steps could be performedconcurrently with each other. Furthermore, if desired, one or more ofthe above described steps may be optional or may be combined withoutdeparting from the scope of the present invention. Furthermore, one ormore of the above described steps may be performed outside the operatingroom to save time spent in the operating room. For example, steps 144and 146 may be performed prior to bringing the patient into theoperating room and prior to step 142.

A technical advantage of this exemplary embodiment for intra-operativehaptic planning of a surgical procedure is that it provides for tightercoupling of the planning and execution phases of the surgical procedure.Planning for the surgical procedure is preferably performedintra-operatively with respect to the patient. Thus, when planning iscomplete, the haptic device is in position for executing the surgicalplan. No additional motion of the haptic device is required to initiatethe execution phase. Furthermore, by using a virtual tool to determinethe trajectory of the real surgical tool to the target region, injury toanatomical features may be avoided during the planning phase.

A haptic object may be of any shape or size. As shown in FIG. 4A, hapticobject 26 may be funnel shaped to guide a medical device, for example asurgical tool, coupled to haptic device 113 toward a target area onanatomy 114 of the patient. The path of the haptic object may depend ona surgical plan. An algorithm may be used to create the funnel shapedhaptic object illustrated in FIG. 4A. The information desired to createthe funnel shaped haptic object may be based on a surgical plan. Ifdesired, haptic object 26 may move with haptic device 113. This allowsguidance of the surgical tool toward the target area from the currentposition of haptic device 113. Thus, the surgical tool may be guidedtoward the target area from any position in proximity to anatomy 114.Furthermore, the surgical tool may be guided from a current pose to adesired pose.

Haptic object 26 may be of any shape, for example, a line, a curve, acylinder, a funnel, and/or the like. Haptic object 26 is, in theillustrated example, defined as a virtual pathway to facilitateinteractive positioning of haptic device 113 and/or surgical tool 112coupled to haptic device 113 at a desired position. Haptic object 26guides surgical tool 112 coupled to haptic device 113 from an initialposition and/or pose toward a target area and/or a desired pose relativeto anatomy 114 of the patient. If desired, haptic object 26 may guidesurgical tool 112 to the target area along a path or trajectory 28. Thepath or trajectory 28 from the initial position to the target area maydepend on the surgical plan. The path may be of any shape, for example astraight line, a curve, a funnel, a cylinder, and/or the like. Based atleast in part on haptic object 26, haptic forces are applied to hapticdevice 113 as the user moves the surgical tool or haptic device to guidethe user in moving the surgical tool 112 along path 28 toward the targetarea.

Haptic object 26 is preferably steerable or reconfigurable. For example,the haptic object may be defined to move or to change position and/ororientation as the haptic device (or the surgical tool or instrumentcoupled to it) moves. This allows, for example, the user to guidesurgical tool 112 toward the target area from almost any position inproximity to anatomy 114. This reconfigurability or steerability ofhaptic object 26 also allows the user to guide surgical tool 112 to thedesired pose from its current position and/or pose.

Haptic object 26 may also be allowed to move from a pre-defined path orposition in order to avoid obstacles, preferably without deviating fromthe target area. This is especially useful in avoiding obstacles in thepath of haptic device 113 that computer-assisted surgery system 11 maynot be aware of. Thus, surgical tool 112 may be steered by the usertoward the target area without colliding with other surgical tools andequipment, the patient, or operating room staff.

Steering, moving or reconfiguring is, in a preferred embodiment, inresponse to application of a force or torque on the haptic device or thehaptic object that exceeds a threshold value. For example, if the userpushes haptic device 113 against the haptic object with a force thatexceeds a threshold, then the haptic object will be repositioned,reconfigured or modified to a new configuration based on the input forceor torque. Preferably, haptic object 26 moves in the direction of theforce or torque thereby providing an intuitive method for repositioningor realigning haptic object 26.

If desired, haptic object 26 may move to a new location if the targetarea is changed. Thus, as shown in FIG. 4A, haptic object 26 may bemoved from an initial position to a new position, as shown by hapticobject 26′, in response to a change in the target area.

In an alternative embodiment, haptic object 26 may be defined as virtuallinear or non-linear springs, dampers, clutches, and/or the like,logically applied to one or more joints of haptic device 113. One ormore joints of haptic device 113 may comprise virtual detentscorresponding to the final desired pose of haptic device 113.Preferably, standard joint-space control techniques are used toimplement the haptic objects at each joint and conventional inversekinematics techniques are used to determine the joint positionscorresponding to the desired Cartesian position/angle of the hapticdevice. The user may avoid obstacles by specifying the sequence in whichthe joints of haptic device 113 “lock” into their detents. The user maybe permitted to modify the selected sequence by “unlocking” jointsduring positioning of surgical tool 112, especially if the sequence isdetermined through a trial-and-error technique. Interactive unlocking ofa joint by the user may be based on the magnitude, duration or dynamicproperty of the force and/or the torque at that joint by the user. Agraphical user interface, a footswitch, a keyboard, a button, and/or thelike, communicatively coupled to haptic device 113 may be used to unlocka joint. If desired, once the desired pose is achieved, the ability tounlock the joints may be disabled to prevent inadvertent motion ofhaptic device 113.

In another alternative embodiment, haptic object 26 may be defined byvirtual linear or non-linear springs, dampers, clutches, and/or thelike, logically associated with one or more redundant degrees-of-freedomof haptic device 113. For example, if a haptic device comprising of fourjoints is used to position the tip of surgical tool 112, then the hapticdevice 113 may be moved along one of the degrees-of-freedom withoutaffecting the position of the tip. Haptic object 26 may be associatedwith the redundant degree-of-freedom to permit the user to interactivelymodify the position of haptic device 113.

FIG. 4B is a flowchart of a method 170 for interactive hapticpositioning of a medical device, for example surgical tool 112 mountedto haptic device 113, using a reconfigurable or steerable haptic object26, all as shown in FIG. 4A. If desired, the reconfigurability of thehaptic object may be user-configurable such that the user may turn thisfeature ON or OFF depending on the application or depending on the stepof a particular application. When the reconfiguration feature isenabled, method 170 is preferably executed periodically.

In step 172, a determination is made as to whether the medical device isin a desired pose. This determination may be made by using sensinginformation from one or more position sensors, such as encoders orresolvers, which may be integrated in the haptic device. If desired,this determination may be made by using sensing information from anexternal device, such as a laser interferometer, a camera, and/or othertracking device.

If in step 172, it is determined that the medical device is in thedesired pose, then in step 174, haptic interaction forces and/or torquesto maintain the pose of the medical device are determined. Thisdetermination may be made based at least in part on the position and/orvelocity of the haptic device and/or the medical device relative to thedesired pose. Any control algorithm now known or later developed may beused for this determination, for example, robust control, adaptivecontrol, hybrid position/force control, Proportional-Derivative (PD)control, Proportional-Integral-Derivative (PID) control, Cartesian basedcontrol, inverse Jacobian control, transpose Jacobian control, and/orthe like. The determined haptic interaction forces and/or torques may betransformed and provided to the haptic device. If in step 172, it isdetermined that the medical device is not in the desired pose, then instep 176, haptic interaction forces and/or torques to maintain themedical device within a haptic object are determined so that the medicaldevice may be guided toward the target area. In step 178, adetermination is made as to whether the result of at least one scalarvalued function of the haptic interaction forces and/or torquescalculated in step 176 exceeds at least one reconfiguration threshold.The reconfiguration threshold may be user-configurable. A scalar valuedfunction computes a value based on one or more input values. In anexemplary embodiment, the scalar valued function may be the square rootof the sum of the squares of the input values. A scalar valued functionmay be applied to one or more haptic interaction forces to provide ascalar value. The resulting scalar value may be compared to thereconfiguration threshold. Dynamic properties of the haptic interactionforces and/or torques, such as direction, duration, and/or the like, mayalso be considered.

If the result of none of the scalar valued functions exceeds thereconfiguration threshold, then the process ends. Otherwise in step 180,haptic object 26 is modified based at least in part on the hapticinteraction forces and/or torques. For example, if the surgeon guidesthe haptic device such that the haptic device in effect pushes againstthe haptic object, the value of the scalar valued function of the hapticinteraction forces and/or torques generated to keep the haptic devicewithin the haptic object may exceed the reconfiguration threshold. Insuch a case, it is desirable that the haptic object be modified, forexample in the direction of the force applied by the surgeon such thatthe surgical tool is maintained within the haptic object. Themodification of the haptic object may comprise changing the size of thehaptic object, changing the shape of the haptic object, pivoting thehaptic object along the target area of the patient's anatomy, and/or thelike.

A technical advantage of this exemplary embodiment for interactivehaptic positioning of a medical device is that by modifying a hapticobject based on the haptic interaction forces and/or torques, greaterflexibility is provided to the surgeon. Thus, the surgeon may approachthe target area without colliding with other surgical tools andequipment, the patient or operating room staff, and still be providedwith haptic cues to enable the surgeon to guide the surgical tool to thetarget area.

The illustrated method for interactive positioning of a haptic deviceusing a reconfigurable (repositionable, steerable) haptic object may beused in any situation where it is desirable to move the haptic device,optionally coupling a component of interest, such as a medical device,for example a surgical tool, and/or the like, within a cluttered orsafety-critical environment. If desired, the haptic device itself may bethe component of interest. The illustrated method may be used in avariety of applications, such as a procedure where virtual constraintsand/or haptic cues are used to move the component of interest into apredefined location and/or orientation and safety or other concerns makeautonomous device motions undesirable. For example, the method may beused in an implant placement procedure, a biopsy procedure, depositionof therapeutic implants, diagnostic palpation of internal or externalanatomy, tumor removal, radiation therapy, artistic or commercialsculpting, artistic or commercial painting, scientific or engineeringexperiments, such as surface digitizing, sample collection, circuitboard probing, manual assembly, fabrication or testing of mechanicaland/or electronic components or assemblies, material handling, and/orthe like.

For rehabilitation and/or physical therapy applications, a haptic devicemay be coupled to the patient using an orthotic device, which mayrequire the patient to grasp a handle. In such an embodiment, the hapticdevice may be coupled to a computer system having a user console. Thecomputer system may or may not be a CAS system, but may be a computersystem designed for rehabilitative or physical therapy applications. Ifdesired, the computer system may be integrated with computer 10. Theorthotic device may have straps, braces, shells, or cast features toprovide a firm or loose connection as desired. The orthotic deviceallows the haptic device to guide, monitor, and/or assist rehabilitativemotions or other exercises. For example, the patient or a therapist maycouple the patient's arm or leg to the haptic device and lead it througha desired motion while the haptic device records the properties of themotion. The motion can then be repeated multiple times without theassistance of the therapist. The haptic device may also be used tomonitor the patient's efforts to move by noticing how much effort isrequired to move the patient, or through the use of force sensingdevices which may be coupled to the haptic device at or near thelocation where the patient interfaces with the haptic device. The hapticdevice may also be used to simply constrain the patient's motion to thedefined path which requires the patient to advance along the definedpath using their own strength. Modes where there is a shared effortbetween the patient and the haptic device may also be advantageous. Itis desirable that when used in this manner, the haptic device operate ina safe manner because it is so close to the patient, who may have onlypartial function in one or more extremities. It may be undesirable forthe haptic device to move to new positions automatically orautonomously. However, it may be desirable to reposition the hapticdevice, for example to permit initial attachment to or grasping by thepatient, so that the haptic device may be moved to different startingpositions between different exercises or repetitions of the sameexercise, or in the course of performing the rehabilitative motions orexercises. A physical therapist may provide the interactive input forrepositioning the haptic device. If desired, the patient may providesuch input while interfacing with the haptic device.

The illustrated method for interactive haptic positioning of a surgicaltool using a reconfigurable or steerable haptic object may beimplemented in software, hardware, or a combination of both software andhardware. The steps discussed herein need not be performed in the statedorder. Several of the steps could be performed concurrently with eachother. Furthermore, if desired, one or more of the above described stepsmay be optional or may be combined without departing from the scope ofthe present invention.

Referring now to FIG. 5, when the user interacts with a haptic object,such as haptic object 20, it is sometimes desirable to know themagnitude of forces applied to the haptic object or the amount that areal or virtual tool or implant is penetrating the haptic object. Fornon-trivial haptic objects, or those with complicated two or threedimensional forms, it may be difficult to present this information in amanner that is simple for the user to understand. However, the desirablepiece of information is often the local penetration distance or hapticrepulsion force. While these can be up to three-dimensional vectorquantities, the magnitude (or length) of such vectors, possibly in thedirection of a local unit normal of the haptic object, are most usefulfor augmenting the haptic interaction of the user. These magnitudes aresimple one-dimensional quantities and can be conveyed to the user in avariety of methods, including meters, dials, numerical displays, graphs,and other visual methods, but also with audio, tactile, haptic, or othermeans.

Though a complete message is conveyed directly by haptic device 113 tothe hand of the surgeon, a visual or audible display can be used tosupport rich interactions between the user and the system. For example,well known and commercially available speech recognition techniques canbe used to provide a verbal method for the user to communicateinformation or instructions to the computer aided surgery system. Speechoutput from the computer aided surgery system 11 can also be used forcommunicating information to the user including status information,warning messages, event notification, and responses to user queries,whether communicated verbally or through some other method. Computermonitors, projection displays, wearable displays, head-mounted displays,stereoscopic views, holographic displays, and/or other visual displaydevices can be used to provide schematic anatomic representations,images of diagnostic datasets, instructions or guides for the surgicalprocedure, depictions of virtual and haptic objects, system statusinformation, patient information, and other information that is easilycommunicated over a visual display. Any other input or output devicecould similarly be used to augment the haptic interaction between theuser and the computer surgery system.

A visual and/or audio display of the penetration into a haptic object ofa predetermined stiffness of a surgical device's depth, force and/orvelocity is provided. The haptic object is based upon information fromthe computer-assisted surgical system. The display is one-dimensional inorder to facilitate the communication of the local penetration magnitudeof the surgical device into the haptic object.

During surgery, the haptic device may be used to enhance the performanceof the surgeon in, for example, such tasks as holding a tool steady,making straight cuts, or moving a tool tip along a path or surface. Thehaptic device can replace mechanical cutting jigs and alignmentapparatus used for aiding in the placement of and preparation of anatomyfor implanted medical devices. Virtual haptic surfaces may be used toreplace physical cutting blocks. The virtual haptic surfaces in thisinstance are preferably software entities that can be easily and cheaplycreated from the models of the implant. The virtual haptic surfaces canbe created with curved shapes, which more closely match the underlyinganatomy and enable implant designs that require less bone or tissueremoval.

Sculpting of a physical object, such as a bone, frequently requiresmultiple planar features to be created in the bone and/or on the surfaceof the bone. A haptic object may be defined to assist in such sculpting.The shape of the defined haptic object may correspond substantially tothe desired resulting shape of the physical object after sculpting. Thephysical object and the haptic object may have segments or surfaces withabrupt transitions and/or may have portions with short radius ofcurvature. As such, it is possible that a surgical tool coupled to thehaptic device and being used to sculpt the physical object may abruptlyfall off one segment causing unintentional damage to the physical objector other objects in the vicinity of the physical object, or bedistracting or disturbing to the user. A segment may be one-dimensional,two-dimensional or three-dimensional.

In order to address this problem, haptic object is dynamically modifiedduring sculpting in order to prevent the surgical tool or the hapticdevice from following an abrupt transition from one segment of thehaptic object to another segment. Preferably, the haptic object remainsin the modified form only so long as it is desirable to prevent abrupttransitioning of the surgical tool or the haptic device from one segmentto another. Once the cutting or portion thereof is complete, the hapticobject may be returned to its original configuration, for example to itsoriginal shape, size, orientation, and/or the like. The modification ofthe haptic object may comprise creating another haptic segment thatprevents the surgical tool from following an abrupt transition from onesegment of the haptic object to another segment of the haptic object,modifying an existing segment of the haptic object, for example byextending the existing segment beyond its boundary, and/or the like.

FIG. 6A illustrates an exemplary haptic device being used for hapticsculpting of physical objects with high curvature. FIG. 6B illustratesan exemplary haptic object 20 for haptic sculpting of physical objectswith high curvature. Referring now to FIGS. 6A and 6B, in certain cases,the desired shape for an anatomical region to be prepared with the aidof a haptic object may include sharp external edges. It is difficult toproperly execute cuts without slipping off these edges, resulting inrounding of the edges and other unwanted artifacts in the resultingcontour of the anatomy. An improved method for preparing these types ofshapes involves dynamically enabling and disabling portions of thehaptic surface. In particular, this method is helpful if a haptic objectcontains at least one sharp external edge where the local angle betweenthe two portions joined by the edge as depicted in FIGS. 6A and 6B isless than 180 degrees. The method includes a way of selecting one ofthese portions, which may include any of the user input modalitiesmentioned herein, but the preferred method is based on proximity to thehaptic object. When one of the portions is selected, that portion of thehaptic object is extended beyond the joining edge to provide acontinuous guide surface. When the extension is no longer required, theuser can return the haptic object to its original configuration bymoving the haptic arm away from the portion or using any other inputmodality.

For example, in a total or unicompartmental knee replacement procedure,multiple planar cuts are often required to prepare the femur for thefemoral implant. A haptic object is defined in software that containsportions closely related to the desired femoral cuts. In experiments,when the user attempts to resect the bone using a cutting burr mountedin the haptic arm using the full haptic object, it is difficult to makethe straight cuts without slipping from one portion to another andfrequently moving the burr beyond the desired region. This slipping mayresult in damage to tendons, blood vessels, ligaments, and otherstructures and distract the user. If instead, each cutting plane of thehaptic object is extended when the user brings the cutting burr withinclose proximity to that portion, it is much easier to create straightcuts without moving beyond the local anatomical site. The portion isreturned to its original extent by simply moving back away from it, atwhich time the user can bring the cutting burr into contact with any ofthe other portions to extend them in a similar manner. While footpedals, voice commands, or other input modalities can be used to controlthe extension of each plane, controlling them in the preferred mannerdescribed previously requires no additional hardware and is extremelysimple for the user. However, a visual display of the haptic object andthe extended portion is also helpful for helping the user to understandmore complex haptic objects, especially where their view of the cuttingis limited due to obstructions or a minimally-invasive technique.

FIG. 6A shows an exemplary system for dynamically extending a hapticobject. A representation of the physical object, for example the anatomyof the patient to be sculpted, may be displayed on display device 30.The representation of the physical object may comprise a two-dimensionalor three-dimensional drawing or image. The image could be, for example,a two-dimensional medical diagnostic dataset or a three-dimensionalmedical diagnostic dataset of the patient. In FIG. 6A, haptic object 20includes two different portions (20′ and 20″) divided by a well definededge 21. When haptic device 113, the surgical tool, or the virtualsurgical tool comes within a predefined distance, say R₁, of oneportion, say portion 20′, that portion of haptic object 20 is activated.If desired, the activated portion of the haptic object 20 may beextended as shown by the broken lines 23 in FIG. 6A. When haptic device113 moves to within a predefined distance of another portion, sayportion 20″, the new portion of haptic object 20 may be activated. Ifdesired, the newly activated portion of haptic object 20 may beextended.

It is desirable that haptic object 20 with high curvature be logicallydivided into or be approximated by a plurality of portions or segmentswithout high curvature. For example, as shown in FIG. 6A, haptic object20 may be logically divided into a plurality of portions 20′ and 20″separated by an edge 21. Although, it is preferable to logically dividea haptic object into a plurality of segments, the haptic object itselfmay be defined using a logical combination of a plurality of segments.For example, a plurality of segments may be initially defined and thehaptic object may be defined as a logical combination of one or more ofthe plurality of segments. If desired, the haptic object may comprise aregular or irregular arrangement of volume elements, or voxels, some orall of which may be labeled. It may be desirable to only label thevoxels on the surface of the object in this manner.

FIG. 6C is a flowchart of a method 120 for dynamically modifying ahaptic object, such as haptic object 20 of FIGS. 6A and 6B. If desired,the dynamic modification feature may be user-configurable such that theuser may turn this feature ON or OFF depending on the application ordepending on the step of a particular application. When the dynamicmodification feature is enabled, method 120 is preferably executedperiodically.

In step 122, a determination is made as to whether a configuration ofthe haptic object, say haptic object 20, has already been modified, forexample by modifying a segment of the haptic object or by adding a newsegment. In the preferred embodiment, the value of a configuration flagmay be checked to determine if haptic object 20 has already beenmodified. If haptic object 20 has not already been modified, then instep 124, a determination is made as to whether one or more criteria formodifying the configuration of haptic object 20 is satisfied. Thecriteria may be proximity of surgical tool 112 coupled to haptic device113 to haptic object 20, penetration of haptic object 20 by surgicaltool 112, gestural motions of surgical tool 112, gestural or othermotion of surgical tool 112 relative to the position of haptic object20, a fixed or variable time period, detection of an unwanted slippageover edge 21, and/or the like. If desired, the criteria may be proximityof the representation of surgical tool 112 to haptic object 20,penetration of the boundaries of haptic object 20 by the representationof surgical tool 112, gestural or other motion of the representation ofsurgical tool 112 relative to the position of haptic object 20, and/orthe like. When modification of the configuration of haptic object 20comprises modifying a segment of haptic object 20, preferably the samecriteria is used to determine if any of the segments should be modified.However, if desired, different segments may be modified based ondifferent criteria. In such an embodiment, each of the plurality ofsegments may have one or more criteria associated with it.

If in step 124, it is determined that at least one criteria formodifying the configuration of haptic object 20 is satisfied, then instep 126, the segment to be modified is selected. Alternatively, asegment in proximity to which a new haptic segment is to be created maybe selected in step 126. In an alternative embodiment, the processstarting at step 126 may be executed if a predefined logical combinationof a set of criteria are satisfied. Preferably, the segment that isclosest to haptic device 113 is selected. However, if desired, othercriteria may be used to select a segment. For example, if surgical tool112 has crossed an edge between two or more segments since the last timemethod 120 was executed, then one of the segments associated with theedge that was crossed may be selected. Alternatively, the segment beingpenetrated by surgical tool 112 may be selected. In step 128, theconfiguration of the selected segment is modified, preferably byextending the selected segment in a desired direction of movement ofhaptic device 113. The configuration flag may be set to indicate thathaptic object 20 has been modified.

The method for modifying the configuration of the selected segment ispreferably based at least in part on the manner in which the hapticobject is represented. This representation may be based on surfacepolygons, voxels, non-uniform rational B-splines (NURBs), constructivesolid geometry, and/or any other method for representing haptic objectsnow known or later developed. The modified segment may be represented inany manner which may or may not be the same as those used to representthe original haptic object. Preferably, the selected segment is extendedsuch that the extended portion is continuous with the segment along oneof its high curvature edges. The extension may be flat or curved. Thesegment may be extended a fixed or variable distance beyond the originalsegment, or could be extended to intersect another portion of the hapticobject or the edge of a workspace. The method used for extending thesegment depends on the method used for representing the extension. Forexample, if a haptic object is represented with surface polygons, thenthe polygons that lie within the segment of interest and adjacent to oneof its boundaries are identified. A neighboring segment that lies beyondthe original segment and has the same normal direction as the originalpolygon may be enabled. For a voxel representation, the voxels may belabeled to indicate whether they behave as solid or filled regions ofspace for configurations of the haptic object with different extendedsegments, which may be automatically, semi-automatically, or manuallydesigned. The selected neighboring segment may be added to the hapticobject. Thus, as illustrated in FIG. 6A, if portion 20′ of haptic object20 is the selected segment, then portion 20′ may be extended beyond itsoriginal boundary, for example as shown by broken lines 23.Alternatively, if desired, a new haptic segment may be created inproximity to the selected segment.

In step 130, haptic interaction forces and/or torques for the hapticobject are calculated. The haptic interaction forces and/or torques maybe transformed and provided to haptic device 113. For example, it may bedesirable to compute appropriate forces and torques for the actuators ofthe haptic device to apply such that the desired haptic interactionforces and/or torques will be produced. In some cases, it may bedesirable to alter position or velocity commands to the actuators toproduce the desired effect. The haptic interaction forces and/or torquesfrom the selected segment may be used to guide haptic device 113 in adesired direction away from, toward, or aligned with physical object 114to be sculpted. The haptic interaction forces and/or torques may berepulsive, attractive, frictional, viscous, impulsive, detent,regulatory (for example designed to maintain cutting speeds or feedrates), and/or the like. If desired, the haptic interaction forcesand/or torques may be calculated using a mathematical, control theory,or machine learning algorithm.

If in step 124, it is determined that the criteria for modifying theconfiguration of haptic object 20 is not satisfied, then the processstarting at step 130 may be executed.

If in step 122, it is determined that the configuration of haptic object20 has already been modified, then in step 134, a determination is madeas to whether one or more predefined criteria for maintaining hapticobject 20 in the modified configuration is satisfied. These criteria mayor may not be the same as those considered when the configuration ofhaptic object 20 was initially modified. Preferably, if at least onecriterion for maintaining the haptic object in the modifiedconfiguration is satisfied, then the process starting at step 130 may beexecuted. Otherwise, in step 136, the modified haptic object is returnedto its original configuration. The configuration flag may be reset toindicate that haptic object 20 has not been modified. After execution ofstep 136, the process starting at step 130 may be executed. In analternative embodiment, the process starting at step 130 may be executedif in step 134 it is determined that a predefined logical combination ofa set of criteria are satisfied.

As illustrated in FIG. 6A, when haptic device 113 or surgical tool 112coupled to haptic device 113 comes within a predefined distance R₁ ofone portion of haptic object 20, say portion 20′, that portion of hapticobject 20 may be activated and modified such that it extends beyond itsoriginal boundary as shown by dashed lines 23. While haptic device 113or surgical tool 112 is in close proximity to portion 20′ or maintainscontact with portion 20′, portion 20′ remains modified. Surgical tool112 may be used during that time to sculpt the portion of physicalobject 114 corresponding to portion 20′ to a desired shape. When thesculpting of the portion of physical object 114 corresponding to portion20′ is completed, the user may move haptic device 113 away from portion20′. Portion 20′ may then be returned to its original configuration.When haptic device 113 or surgical tool 112 moves to within a predefineddistance of another portion of haptic object 20, say portion 20″,portion 20″ of haptic object 20 may be activated and modified such thatit extends beyond its original boundary.

The illustrated method for dynamically modifying a haptic object may beused in a variety of applications, such as any procedure where a virtualconstraint and/or haptic cues are used to guide a user using a hapticdevice for sculpting a physical object or shape that has high curvature.For example, the method may be used in fabrication of components forconsumer or industrial products, for the reproduction or creation ofartistic pieces, such as sculptures, for shaping bones in an orthopedicprocedure, and/or the like.

The illustrated method for dynamically modifying a haptic object may beimplemented in software, hardware, or a combination of both software andhardware. The steps discussed herein need not be performed in the statedorder. Several of the steps could be performed concurrently with eachother. Furthermore, if desired, one or more of the above described stepsmay be optional or may be combined without departing from the scope ofthe present invention.

A technical advantage of this exemplary embodiment for dynamicallymodifying a haptic object is that the sculpting of the physical objectmay be performed in a more controlled manner. Thus, during a surgicalprocedure, unintentional damage to parts of the body may be avoided andthe user can feel more comfortable using the system. Another technicaladvantage is that the user does not have to move its attention away fromthe working volume when switching from one segment to another segment ofthe haptic object. Yet another technical advantage is that shapes withhigh curvature may be operated on more easily than if only the entirehaptic object were used.

FIG. 8 illustrates the use of an exemplary haptic device 113 as an inputdevice. Haptic device 113 and a haptic object 20 in real space areillustrated. Haptic device 113 may also be used as an input device,allowing information to pass from the user to CAS system 11, andproviding functionality similar to common user interface devices, suchas a mouse, touchpad, keyboard, joystick, flight controller, hapticjoystick, or any other input device. When used as an input device, itmay be used for defining anatomical reference geometry, manipulating theposition and/or orientation of virtual implants, manipulating theposition and/or orientation of surgical approach trajectories,manipulating the position and/or orientation of bone resections, and theselection or placement of any other anatomical or surgical feature.Haptic device 113 may also be used for more generic user interfacefunctions, including but not limited to, moving a cursor 31 (FIG. 8),selecting buttons or other similar user interface objects, selectingpull-down menus, manipulating on-screen dials, knobs, and othercontrols. When in this user-input mode the haptic device can beconstrained to move in only certain directions which may be definedrelative to the position of a predetermined portion of the hapticdevice, relative to the position of the patient or a portion of thepatient anatomy, or relative to images or 3-D models of schematic,virtual, atlas, or actual patient anatomical features. The predeterminedportion of the haptic device may be capable of moving.

As illustrated in display 30 of FIG. 8, haptic device 113 may be used asan input device to change the position, shape, size, etc. of hapticobject 20. An example of an application of haptic device 113 used inthis mode is planning the placement of a knee implant. After acquiringappropriate anatomical images of the anatomy of interest, the computersurgery system enters a mode where a cursor appears on a display visibleto the user. The user grasps the arm to move the position of the cursor,possibly in multiple views. When satisfied with the position of thecursor, the user fixes it in the desired position through the use of afoot pedal, button, wired or wireless control pendant, voice command, orother input, or through the application of a force or torque to thehaptic arm, or moving the haptic arm in a distinctive gesture, such as atap, twist, or other gesture that is easily distinguishable from theuser interactions during the cursor positioning. After the firstposition is set, a second cursor is used to define the endpoint of aline connecting to the two or three-dimensional position of the firstcursor. The second cursor is moved, as above, to define an anatomicalaxis of the femur bone and its position is fixed using one of the abovementioned techniques. The two or three dimensional position andorientation of the implant can then be manipulated by the user using thehaptic device as an input device. The implant is constrained by thesystem such that one of its surfaces is perpendicular to the anatomicalreference line, but its position and orientation can be adjusted by theuser. It is also possible to allow deviations from the anatomical axis,possibly coupled with displays of such deviations relative to anatomicalreference frames familiar to the user. For example, the varus/valgusangle of the implant relative to the anatomical reference line can beadjusted and displayed to allow appropriate alignment of the kneeimplants. This general technique can be adapted to plan the approachand/or placement of minimally invasive hip and knee implants, traumafixation pins, pedicle screws, biopsy needles, radioactive beads,radiotherapy beam emitter, or any other medical device.

With a haptic device, the surgeon can use tools identical or verysimilar to those used in standard practice. By exploiting the hapticfeatures of the device, the need for awkward teach pendants or GUI-basedrobot controls may be reduced or eliminated. Switching between freehandand assisted steps of a procedure is quickly performed by simply pushingthe device out of the way, similar to familiar operating room objectssuch as microscopes and overhead lights. While the systems may beinternally complex, the surgeon must be shielded from this complexity sothat he can focus all of his attention on his patient.

For example, the haptic arm can hold itself at a reference positionusing a joint-based or Cartesian control algorithm. The user appliesforces and/or torques to the arm, either on an interaction handle orend-effector or at any point on the arm, which cause the arm to deflectfrom the reference position. The amount and direction of the deflectionis continuously communicated to the computer system to modify theposition of any desired virtual reference geometric feature or userinterface object.

In another example, the haptic arm can hold itself at a referenceposition using a joint-based or Cartesian control algorithm but with twodegrees of freedom left unconstrained. The user can then move the arm inthe unconstrained directions to provide two-dimensional control of auser-interface object, such as a cursor, implant, or other geometric orvirtual surface entity. A similar technique can be used for one degreeof freedom manipulation of objects, such as user interface slider bars,implant lengths, positions of objects along a reference trajectory, orany other one-dimensional control such as audio volume, imagebrightness, object scaling, image zooming, and the like. A similartechnique can be used for higher than three degree of freedompositioning of implants or virtual or haptic objects. The hapticobject's position may also be constrained relative to any relevantanatomical features for a particular application. For example, a kneeimplant may be constrained to have the proper alignment relative to theanatomical axis of the leg, or to achieve proper ligament balance, butwith the other degrees of freedom controllable by the user in the mannerdescribed above.

The stiffness or damping of the control algorithm may vary in differentdirections to indicate preferential directions of motion which may bealigned with any direction as described in the previous paragraph. Thisstiffness variation may include zero stiffness along certain directionsor may lock the user to the preferred directions once the deviation fromthe reference position exceeds some threshold value. This stillnessvariation assists with simplifying the planning process by allowing theuser to focus their attention on a limited number of degrees of freedomat a time. For example, the user may set the position of an implantalong one or two directions first, then set the position of the implantalong an additional direction or directions without disturbing the setdirections.

The stiffness and damping variations can occur automatically dependingon the physical interaction of the user with the haptic device and doesnot require the use of another input device such as a voice command,control pendant, or foot pedal. Any such simplification has benefits inreducing service costs, simplified system use, and improved safety. Thisgeneral method of planning also allows the surgeon to perform planningwithout having to leave the normal operating position to interact withthe computer-aided surgery system or requiring an assistant to controlthe computer-aided surgery system or requiring the introduction ofadditional input devices other than the haptic device which is alreadybeing used for execution of the surgical plan. An additional benefit ofthis use of a haptic device is that the motion of the controlled objectcan be scaled relative to the motion of the arm, so that it can bepositioned to a precision better than the user can position a realobject, eliminating the deleterious effects of the user's hand tremorand any force disturbances arising from friction, backlash, magneticdetent forces, and other force disturbances arising from the haptic arm.It should be noted that the primary function of the object controlled bythe haptic device is something other than monitoring the pose of thehaptic device or monitoring the pose of a component of interest that mayor may not be coupled to the haptic device.

FIGS. 7A and 7B illustrate the use of a haptic device and a surgicaltool to define a haptic object. In the illustrated example, hapticdevice 113 is being used as an input device to define haptic object 182.In order to use haptic device 113 as an input device to define hapticobject 182, the user grasps surgical tool 112 coupled to haptic device113. If desired, the user may grasp haptic device 113 itself. Usingsurgical tool 112 the user traces the boundaries of a desired region,for example a portion of the anatomy with respect to which the hapticobject is to be defined. The user may trace the boundary, for example bytouching the end of surgical tool 112 to portions of the desired regionof the anatomy. The motion of surgical tool 112 may be recorded and thelocations of the endpoints traced by the user computed. The geometryand/or location of haptic object 182 may be determined based at least inpart on the location of the endpoints. A haptic device creation mode maybe used to specify the desired shape of the haptic object. For example,to create a cylindrical haptic object that corresponds to a resectedportion 184 of anatomy 114, the user can trace a plurality of points onthe boundary of resected portion 184. An appropriate cylindrical hapticobject may be created using any technique now known or later developed.

Material and other properties of the anatomy may be defined by probingthe anatomy. For example, surgical tool 112 may include a forcemeasurement device coupled to the tip of surgical tool 112.Alternatively, if desired, instead of surgical tool 112, a probecomprising a force measurement device may be coupled to haptic device113. When the user interfaces the force measurement device against aportion of anatomy 114, the force may be measured by the forcemeasurement device. The measured force may be displayed as a function ofthe distance the anatomy moves, if any, upon application of the force.The stiffness of that portion of anatomy 114 may be calculated as theratio of the force to the distance. If desired, haptic device 113 itselfmay be interfaced with a portion of anatomy 114 and the force determinedbased on the torques provided by the actuators. In such an embodiment,haptic device 113 may make small or large movements or press againstportions of anatomy 114 in an autonomous mode without any physicalassistance from the user. The force may be determined using any Jacobianmethod now known or later developed. The graphical representation 186 ofFIG. 7B illustrates the force with which surgical tool 112 comes incontact with anatomy 114 as a function of displacement of anatomy 114.

If desired, other types of sensing devices may be coupled to hapticdevice 113 or surgical tool 112 to determine other properties of anatomy114. These properties may be used to determine the type of tissue thatis in proximity to haptic device 113. Thus, haptic device 113 may beused to differentiate between hard and soft bones, healthy and diseasestissues, different types of healthy tissues, boundaries of anatomicalstructures, etc. Based on information received from haptic device 113,the type of the tissue may be automatically determined by CAS system 11and displayed on display device 30.

FIG. 9 is a flowchart of a representative method 190 for using hapticdevice 113 as an input device. In step 192, the input mode is initiated.The user may initiate the input mode by any mechanism now known or laterdeveloped. For example, the user may use a graphical user interface, afootswitch, a keyboard, a button, and/or the like, to indicate that theuser desires to use haptic device 113 as an input device. Haptic device113 may control a plurality of objects. However, it is desirable that itonly control a single object at one time. As such, in step 194, anidentification of an object to be controlled is received. The controlledobject may be a cursor, a button, an on-screen dial, a knob, a sliderbar, or other similar user interface object, a virtual implant, asurgical approach trajectory, a bone resection, and/or the like. Theuser may select the object to be controlled by any method now known orlater developed, for example by selecting the object using aconventional input device.

In step 196, a reference pose for haptic device 113 may be stored. Thereference pose is preferably the current pose of haptic device 113. Forexample, in this step, position information about the tip of hapticdevice 113 may be stored. In step 198, the controlled object iscorrelated with haptic device 113. The correlation of the controlledobject with haptic device 113 is desirable so that movement of hapticdevice 113 may be translated or mapped into a corresponding movement oraction relative to the controlled object. The correlation or mappingallows a determination of the amount or direction of movement of thecontrolled object in response to movement of haptic device 113. Forexample, the user may specify that movement of haptic device 113 by oneunit should cause a controlled object, for example cursor 31, to move byten pixels on display device 30.

The user may move haptic device 113 around to control the objectselected in step 194. In step 200, a change in pose of haptic device 113is determined. The change in pose of haptic device 113 is preferablydetermined relative to the reference pose of haptic device 113. Thechange in pose of haptic device 113 may comprise, for example, a changein position of the tip of haptic device 113.

In step 202, the reference pose of haptic device 113 may be updated.Preferably, the reference pose is updated based at least in part on thechange in pose of haptic device 113. If desired, the reference pose maybe updated based at least in part on a wrench applied to haptic deviceby the user. The wrench may be explicitly measured by a sensor. Ifdesired, the wrench may be implicit in that the haptic device candetermine that a wrench is being applied.

In step 204, new parameters for the controlled object are calculated.The parameters of the controlled object may be, for example its pose,position, angle, size, color, shape, orientation, view direction,brightness, contrast, table indices, status, mode, configuration, and/orthe like. The new parameters may be calculated based on the change inpose of haptic device 113 and/or the wrench applied to haptic device bythe user. If desired, the new parameters may be calculated based on thechange in reference pose of haptic device 113. Preferably, correlationinformation obtained in step 198 is used to calculate the newparameters. The new parameters may be used to change the controlledobject. Thus, for example, when the controlled object is cursor 31 andthere is a change in pose of haptic device 113, then a new pose for thecontrolled object may be determined based on the new parameters. In step206, the controlled object is changed based on the new parameters. Thus,for example, if the controlled object is cursor 31, then the position ofcursor 31 on display device 30 may be changed based at least in part onthe new parameters calculated in step 204.

In step 208, a haptic wrench applied by the haptic device to the medicaldevice and/or the user is determined. The haptic wrench may bedetermined based on the new parameters of the controlled object, thechange in pose of haptic device 113, and/or the current pose of hapticdevice 113.

In step 210, the determined haptic wrench is applied to haptic device113. Instead of allowing haptic device 113 to be moved in any direction,it may be desirable to constrain the movement of haptic device 113. Thedetermined haptic wrench when applied to haptic device 113 prevents itfrom moving in certain undesirable directions. For example, if thecontrolled object is capable of moving in only one dimension, it may bedesirable to constrain the motion of haptic device 113 so that hapticdevice 113 moves in only one direction. As another example, when theobject being controlled is cursor 31 on display device 30, then it maybe desirable to constrain the movement of haptic device 113 to atwo-dimensional plane corresponding to display device 30. As a furtherexample, if it is not desirable for haptic device 113 to move largedistances from the reference pose, the haptic wrench may act to returnhaptic device 113 to the reference pose in one or more directions.

Within the input mode, haptic device 113 may be used in a positioncontrol mode or a rate control mode. In the position control mode, thechange in pose of the controlled object tracks the change in pose ofhaptic device 113. For example, if haptic device 113 is moved in aparticular direction by one unit, the controlled object moves in acorresponding direction by a corresponding amount. When haptic device113 is released, it stays in its new pose.

On the other hand, in the rate control mode, the displacement of hapticdevice 113 from the reference pose and/or the wrench applied to thehaptic device by the user, may control the velocity of the controlledobject. For example, if haptic device 113 is maintained in its referencepose (or if no wrench is applied to the haptic device by the user), thenthe rate of movement of the controlled object is zero. The displacementof haptic device 113 from the reference pose (or the magnitude of thewrench applied by the user to the haptic device) determines the velocityof movement of the controlled object with the velocity of movement beingproportional to the displacement of the controlled object (or to themagnitude of the wrench applied to the haptic device). When it isdesirable to move the controlled object, haptic device 113 is simplymoved (or pushed) in the direction of the desired motion of thecontrolled object. When haptic device 113 is released it moves back tothe reference pose due to application, in step 210, of the haptic wrenchdetermined in step 208. Thus, in the rate control mode, the controlledobject may be moved a substantial distance without substantially movinghaptic device 113.

In step 212, a determination is made as to whether haptic device 113 isstill operating in the input mode. If haptic device 113 is not operatingin the input mode, then the process terminates. Otherwise, in step 214,a determination is made as to whether a new object to be controlled hasbeen specified. If a new object to be controlled has not been specifiedthen the process starting at step 200 to determine the change in pose ofhaptic device 113 may be executed. Otherwise, the process starting atstep 194 to receive identification of the new object to be controlled isexecuted.

For example, in one embodiment, the reference pose may be associatedwith the desired trajectory of a drill guide attached to haptic device113. In such an embodiment, updating the reference pose in step 202comprises changing the desired trajectory of the drill guide. When theuser moves haptic device 113 from the reference pose for a prolongedperiod of time, the reference pose will be updated to move in thedirection of the user's deflection. If, in step 210, an appropriatehaptic feedback wrench is applied, then upon release of haptic device113 by the user, haptic device 113 will assume the new reference pose.When the user is satisfied with the reference pose and the input mode isterminated in step 212, haptic device 113 will be in a pose such thatthe drill guide is aligned with the desired trajectory.

The illustrated method for using a haptic device as an input device maybe implemented in software, hardware, or a combination of both softwareand hardware. The steps discussed herein need not be performed in thestated order. Several of the steps could be performed concurrently witheach other. Furthermore, if desired, one or more of the above describedsteps may be optional or may be combined without departing from thescope of the present invention.

A technical advantage of using a haptic device as an input device in themanner described above is that the use of an additional input device maybe avoided thereby reducing the clutter in the operating room.

1-50. (canceled)
 51. A haptic surgical system, comprising: acomputer-aided surgery system; a haptic device coupled to a surgicalinstrument and said computer-aided surgery system; and a processorprogrammed to: define a virtual surface to which a target region of ananatomy of a patient is to be sculpted using the surgical instrument;generate a haptic interaction wrench based at least in part on a currentpose of said surgical instrument relative to the virtual surface toguide a user in moving said surgical instrument along the virtualsurface when sculpting the anatomy of the patient.
 52. (canceled) 53.The system of claim 52, wherein said virtual surface is a curvedsurface.
 54. The system of claim 51, wherein said processor is furtherprogrammed to provide at least one haptic cue to guide said user inmoving said surgical instrument from the current pose to a desired pose.55. The system of claim 54, wherein said processor is further programmedto provide the haptic cue to assist the user in sculpting a smoothsurface without abrupt transitions on the anatomy of the patient. 56.(canceled)
 57. The system of claim 51, wherein said processor is furtherprogrammed to determine at least one haptic interaction wrench to beapplied to said haptic device to maintain said haptic device in thecurrent pose.
 58. (canceled)
 59. The system of claim 51, wherein saidprocessor is further programmed to provide at least one haptic cue toguide said user in moving said haptic device toward said target regionalong a path defined at least in part by said virtual surface.
 60. Asurgical method, comprising: defining a virtual pathway to a targetregion of an anatomy of a patient on whom a surgical procedure is to beperformed and a virtual boundary defining an anatomical structure of thepatient to be avoided; storing the defined virtual pathway and thedefined virtual boundary; manually using a surgical instrument attachedto a haptic feedback device to perform a surgical procedure on thepatient such that a surgeon is provided with direct tactile feedbackfrom interaction between the surgical instrument and the anatomy of thepatient; supplementing the direct tactile feedback with (1) virtualpathway haptic feedback to the surgeon indicative of interaction betweenthe surgical instrument the stored virtual pathway and (2) protectivehaptic feedback to the surgeon indicative of interaction between thesurgical instrument and the virtual boundary.
 61. The method of claim60, further including: modifying at least one of said virtual pathwayand the virtual boundary in response to a value of a scalar valuedfunction of a haptic interaction wrench applied to said haptic deviceexceeding at least one threshold value.
 62. The method of claim 60,further including: modifying at least one of the virtual pathway and thevirtual boundary in response to the surgeon pushing said surgicalinstrument against at least one of the virtual pathway and the virtualboundary during the surgical procedure and applying a haptic interactionwrench in response to said pushing.
 63. The method of claim 61, furthercomprising: modifying at least one of the virtual pathway and thevirtual boundary in response to a value of a scalar valued function ofsaid haptic interaction wrench exceeding at least one threshold value.64. The method of claim 63, wherein said at least one threshold value isuser configurable.
 65. The method of claim 60, wherein at least one ofthe virtual pathway haptic feedback and the protective haptic feedbackprovide a haptic wrench which urges the surgeon to position the surgicalinstrument in a desired pose relative to said target region.
 66. Themethod of claim 60, further comprising: guiding said surgeon to movesaid surgical instrument into a desired pose relative to said targetregion from a current pose in accordance with at least one of thevirtual pathway haptic feedback and the protective haptic feedback. 67.The method of claim 60, wherein the haptic feedback includes: a hapticinteraction wrench which (1) attracts said surgical instrument toward atleast one of the virtual pathway and the virtual boundary to maintainsaid surgical instrument in a desired pose.
 68. The method of claim 60,wherein defining the virtual pathway includes defining a virtual cuttingsurface and further including: dynamically enabling and disablingportions of the virtual cutting surface to facilitate smoothlytransitioning from one portion of the cutting surface to another portionof the cutting surface without leaving sharp edges.
 69. The method ofclaim 60, wherein the surgical procedure is a joint replacement and thevirtual pathway defines at least one virtual cutting surface and furthercomprising: tracking and displaying at least one of a current positionand a current orientation of the surgical instrument on the display;modifying the virtual cutting surface in accordance with the at leastone of the current position and the current orientation of the surgicalinstrument relative to the cutting surface.
 70. The method of claim 60,wherein the surgical instrument includes a bone removal surgicalinstrument for an orthopedic surgical procedure on a bone of the patientand the virtual pathway defines at least one virtual cutting surface,and further including: dynamically modifying the virtual cutting surfacebased on a current position of the bone removal surgical instrumentrelative to the virtual cutting surface to avoid abrupt transitions suchthat a bone surface is sculpted without sharp edges.
 71. A method ofperforming haptic guided computer assisted surgery on a joint of apatient, the method comprising: displaying a diagnostic image of thepatient's joint; defining at least one virtual surface in a targetregion of the patient's joint; during the surgery, moving a manual inputdevice to move a bone cutting tool along the virtual surface in thetarget region of the patient's joint to remove bone, applying hapticfeedback to the manual input device indicative of a relationship of thebone cutting tool to the virtual surface to guide a surgeon using themanual input device to move the bone cutting tool along the virtualsurface, and modifying the haptic feedback based on a current positionand orientation of the bone cutting tool to guide the surgeon to movethe bone cutting tool from the current position and orientation to adesired position and orientation relative to said target region.
 72. Themethod of claim 71, wherein said modifying step includes: modifying ahaptic interaction wrench applied to the manual input device such thatthe bone cutting tool smoothly moves along the virtual surface.
 73. Themethod of claim 71, further including: providing protective hapticfeedback to the manual input device indicative of an interaction of thebone cutting tool and a virtual boundary defining an anatomicalstructure of the patient's joint to be avoided, and wherein the hapticfeedback is further applied in accordance with a relationship betweenthe bone cutting tool and the virtual boundary.
 74. The method of claim71, wherein said modifying step comprises: modifying said virtualsurface in response to the surgeon pushing the manual input device. 75.The method of claim 74, further comprising: generating at least onehaptic interaction wrench in response to said pushing of said manualinput device to move the bone cutting tool against said virtual surface;and modifying the virtual surface in response to a value of a scalarvalued function of the haptic interaction wrench exceeding aconfigurable threshold value.
 76. The method of claim 71, whereinapplying the haptic feedback includes: generating a haptic wrench on themanual input device which inhibits the bone cutting tool from crossingthe virtual surface.
 77. The method of claim 71, further comprising:providing a haptic cue to guide the surgeon in moving the manual inputdevice to move said bone cutting tool relative to said target regionalong a path defined at least in part by said virtual surface withoutsudden transitions.
 78. The method of claim 71, further comprising:determining a next desired pose for said bone cutting tool when the bonecutting tool is in a current pose relative to said target region; andapplying the haptic feedback to encourage the surgeon to move the bonecutting tool from the current pose to the determined next desired pose.79. A system for haptic positioning of medical device coupled with ahaptic device during a medical procedure, the system comprising: adisplay which displays a diagnostic image of an anatomy of a patientcontaining a target region in which the medical procedure is to beperformed using the medical device; a tracking system which tracks apose of the medical device relative to the target region; a processorprogrammed to: define a virtual pathway based on user input informationdepicting a proposed pathway to be followed by the medical devicerelative to the target region of the anatomy of the patient, control thedisplay to display a visual representation of the virtual pathway and acurrent pose of the medical device on the displayed image of theanatomy; and a haptic feedback system which provides haptic feedback tothe haptic device indicative of (1) an interaction between the medicaldevice and a region of anatomy of the patient to be avoided and (2) arelationship of the medical device to the virtual pathway.
 80. Thesystem of claim 79, wherein said processor is further programmed to:modify said virtual pathway in response to at least one of the currentpose of the medical device and a user of said haptic device pushing saidhaptic device to push said medical device against said virtual pathway;and control the haptic device to generate a haptic interaction wrench inresponse to at least one of the current pose and the pushing of saidmedical device against said virtual pathway.
 81. The system of claim 79,wherein said processor is further programmed to: modify said virtualpathway in response to a value of at least one scalar valued function ofa haptic interaction wrench responsive to pushing the medical deviceagainst the virtual pathway in excess of a threshold value, but notresponsive to a haptic interaction wrench generated in response topushing the medical device against the anatomy of the patient.
 82. Thesystem of claim 79, wherein the medical device is a bone cuttingsurgical instrument and the virtual pathway defines a cutting planeindicating a surface of a bone to be left after portions of bone areremoved by the bone cutting surgical instrument.
 83. The system of claim79, wherein said processor is further programmed to: control the hapticdevice to provide at least one haptic cue based on said virtual pathwayand the current pose of the medical device to guide a user to apreselected next pose of the medical device.
 84. The system of claim 79,wherein said processor is further programmed to: determine a hapticinteraction wrench to be applied to said haptic device to maintain saidmedical device in a desired pose; and control the haptic feedback systemto apply the determined haptic interaction wrench.
 85. The system ofclaim 79, wherein the medical device is a bone removal surgicalinstrument, the virtual pathway defines a target bone surface to be leftafter a portion of bone is removed, and said processor is furtherprogrammed to: dynamically modify the virtual pathway based on a currentrelationship between the bone removal surgical instrument coupled to thehaptic device and the target bone surface to inhibit sharp transitionsin the bone surface left after bone removal.